{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/bleu_evaluation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tcSolREOl8Gl"
   },
   "source": [
    "<div align=\"center\">\n",
    "<h1><a href=\"https://github.com/peremartra/Large-Language-Model-Notebooks-Course\">Learn by Doing LLM Projects</a></h1>\n",
    "    <h3>Understand And Apply Large Language Models</h3>\n",
    "    <h2>Evaluating translations with BLEU</h2>\n",
    "    by <b>Pere Martra</b>\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "\n",
    "<div align=\"center\">\n",
    "    &nbsp;\n",
    "    <a target=\"_blank\" href=\"https://www.linkedin.com/in/pere-martra/\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>\n",
    "    \n",
    "</div>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hVAkT7wBZjhZ"
   },
   "source": [
    "In this notebook, we will use the BLEU metric to compare the quality of two different approaches for performing translations.\n",
    "\n",
    "As my primary language is Spanish, I will translate a few lines from the beginning of this chapter from English to Spanish. My translations will be taken as the reference translations. In other words, they will be used as the basis upon which the quality of the automatic translations will be determined.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BDp4f1CQp3hS"
   },
   "outputs": [],
   "source": [
    "#Sentences to Translate.\n",
    "sentences = [\n",
    "    \"In the previous chapters, you've mainly seen how to work with OpenAI models, and you've had a very practical introduction to Hugging Face's open-source models, the use of embeddings, vector databases, and agents.\",\n",
    "    \"These have been very practical chapters in which I've tried to gradually introduce concepts that have allowed you, or at least I hope so, to scale up your knowledge and start creating projects using the current technology stack of large language models.\"\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KwvAUtyGbNQ8"
   },
   "outputs": [],
   "source": [
    "#Spanish Translation References.\n",
    "reference_translations = [\n",
    "    [\"En los capítulos anteriores has visto mayoritariamente como trabajar con los modelos de OpenAI, y has tenido una introducción muy práctica a los modelos Open Source de Hugging Face, al uso de embeddings, las bases de datos vectoriales, los agentes.\"],\n",
    "    [\"Han sido capítulos muy prácticos en los que he intentado ir introduciendo conceptos que te han permitido, o eso espero, ir escalando en tus conocimientos y empezar a crear proyectos usando el stack tecnológico actual de los grandes modelos de lenguaje.\"]\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VQ3Wm2Z9gCTf"
   },
   "source": [
    "We will perform the first translation using the NLLB model, a small model specialized in performing translations, which we will retrieve from Hugging Face."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DTO1tNuMrPn-"
   },
   "outputs": [],
   "source": [
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline\n",
    "model_id = \"facebook/nllb-200-distilled-600M\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nDlw6xChiHf4"
   },
   "source": [
    "When creating the pipeline, we pass the source language and the target language of the translation to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SEQT4ae9R2M5",
    "outputId": "8fbdadf1-6b60-402d-cf01-876bc32479dc"
   },
   "outputs": [],
   "source": [
    "translator = pipeline('translation', model=model, tokenizer=tokenizer,\n",
    "                        src_lang=\"eng_Latn\", tgt_lang=\"spa_Latn\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2y7br7o0R3mS",
    "outputId": "a22b18b3-34fb-4d2c-a3a5-cc2cbfdd9aca"
   },
   "outputs": [],
   "source": [
    "translations_nllb = []\n",
    "\n",
    "for text in sentences:\n",
    "  print (\"to translate: \" + text)\n",
    "  translation = \"\"\n",
    "  translation = translator(text)\n",
    "\n",
    "  #Add the summary to summaries list\n",
    "  translations_nllb += translation[0].values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G4mfFiNSiZlN"
   },
   "source": [
    "Now we have the translations stored in the list 'translations_nllb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vGS0JZ0GWMe-",
    "outputId": "6ba03781-ca31-4ffe-922c-b25547a22488"
   },
   "outputs": [],
   "source": [
    "translations_nllb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QmtO4nZemvRK"
   },
   "source": [
    "##Create Translations with Google Traslator.\n",
    "\n",
    "As a second source for translations, we will use the Google Translator API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XeeOjqI9Uovs",
    "outputId": "c050b36a-1d8d-47ee-9c7f-9e393e5179b6"
   },
   "outputs": [],
   "source": [
    "!pip install -q googletrans==3.1.0a0\n",
    "from googletrans import Translator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RNxy4ql9P-ow"
   },
   "outputs": [],
   "source": [
    "translator_google = Translator()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FAJvQQ_ZQd4i",
    "outputId": "de5f202f-787f-4971-e2d9-c9cd05cb3098"
   },
   "outputs": [],
   "source": [
    "translations_google = []\n",
    "\n",
    "for text in sentences:\n",
    "  print (\"to translate: \" + text)\n",
    "  translation = \"\"\n",
    "  translation = translator_google.translate(text, dest=\"es\")\n",
    "\n",
    "  #Add the summary to summaries list\n",
    "  translations_google.append(translation.text)\n",
    "  print (translation.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m0M4Z5cUqC5z"
   },
   "source": [
    "In this list, we have the translations created by Google."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9EkycPnUUL0m",
    "outputId": "61bfa17f-111d-4076-9bee-b133546d37a1"
   },
   "outputs": [],
   "source": [
    "translations_google"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jXuT5sl8ptCo"
   },
   "source": [
    "## Evaluate translations with BLEU\n",
    "\n",
    "We will use the BLEU implementation from the Evaluate library by Hugging Face."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UYNwyeY-p5Ft"
   },
   "outputs": [],
   "source": [
    "pip install -q evaluate==0.4.1\n",
    "import evaluate\n",
    "bleu = evaluate.load('bleu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lRAYkTNDp2qg"
   },
   "outputs": [],
   "source": [
    "results_nllb = bleu.compute(predictions=translations_nllb, references=reference_translations)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PSkAJAfWq0SS"
   },
   "source": [
    "To obtain the metrics, we pass the translated text and the reference text to the BLEU function.\n",
    "\n",
    "Note that the translated text is a list of translations:\n",
    "[\"Translation1\", \"Translation2\"]\n",
    "\n",
    "Whereas the reference texts are a list of lists of text. This allows for providing multiple references per translation:\n",
    "\n",
    "[[\"reference1 Translation1\", \"reference2 Translation1\"],\n",
    "[\"reference2 Translation2\", \"reference2 Translation2\"]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qrvknChXUGPv"
   },
   "outputs": [],
   "source": [
    "results_google = bleu.compute(predictions=translations_google, references=reference_translations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dop9U2ds2YKE",
    "outputId": "89225e04-1c28-4534-f367-caf373969a23"
   },
   "outputs": [],
   "source": [
    "print(results_nllb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "329dBCZxT0F4",
    "outputId": "f7812ae8-0a5f-4ae8-9df0-5f1f3cd7e094"
   },
   "outputs": [],
   "source": [
    "print(results_google)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lZgXkAq3r4mz"
   },
   "source": [
    "It appears that the translation performed by the Google API is significantly better than the one performed by the NLLB model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5fvxCA0rr54W"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyO+p565bsEqwtEDLmIW6e/O",
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
